Move reference results metadata into each archive#882
Conversation
Store generation metadata inside each archive's case directory instead of writing one shared file per tutorial.
PranjalManhgaye
changed the title
github-project-automation
Bot
moved this to Planned next
in GSoC 2026: System tests improvements
PranjalManhgaye
moved this from Planned next
to Needs review
in GSoC 2026: System tests improvements
| - /runs/{{ tutorial_folder }}/{{ precice_output_folder }} | ||
| - /runs/{{ tutorial_folder }}/{{ reference_output_folder }} | ||
| - "-rtol {{ tolerance }} --ignore-missing-reference-files --diff" | ||
| - "-rtol {{ tolerance }} --ignore-missing-reference-files --ignore-unsupported-file-formats --diff" |
There was a problem hiding this comment.
I thought we already had this - good catch!
There was a problem hiding this comment.
Thanks, While re-checking this with a real fieldcompare install, I found that --ignore-unsupported-file-formats alone is not enough for the embedded metadata: fieldcompare fails earlier with "Missing source file" because the metadata exists only in the unpacked reference directory.
I added --ignore-missing-source-files as well. Locally verified that this skips the metadata file while still catching real field regressions.
There was a problem hiding this comment.
Actually, wait: Why do we need fieldcompare while generating the reference results?
There was a problem hiding this comment.
yess ofc fieldcompare is not used during generation, It's needed later, when system tests compare against the new archives: the metadata is only in the unpacked reference dir, so fieldcompare would fail with "Missing source file" without that flag.
There was a problem hiding this comment.
Right, I forgot about it. However, I am afraid that the combination --ignore-missing-reference-files --ignore-missing-source-files will report false success in cases where the solver generated results at different times.
Could we instead switch fieldcompare to only check specific file extensions? The fieldcompare --file-type might be the right option to use.
There was a problem hiding this comment.
Something looks wrong: In this run, I see the updated template in the tools/ directory, but I cannot find the updated reference-results.metadata in it.
While we are on it, let's rename the file to reference-results-metadata.txt, to make it easier to open it with editors.
There was a problem hiding this comment.
Thanks for testing the generation run. I checked the regenerated archive from that run: the updated metadata is already inside the case folder of the .tar.gz (new format, dated 2026-07-18).
The shared reference-results/reference_results.metadata outside the archive is a leftover from before this PR and was not refreshed because we stopped writing it.
Also renamed the embedded file to reference-results-metadata.txt (and the template accordingly), as suggested.
There was a problem hiding this comment.
In the archive quickstart_fluid-openfoam-solid-cpp_2026-07-18-092817/reference-results/fluid-openfoam_solid-cpp.tar.gz, I do not see any metadata file. Should I look elsewhere?
Rename the per-archive metadata file to reference-results-metadata.txt and add --ignore-missing-source-files so fieldcompare does not fail on the metadata when comparing against regenerated archives.
2 participants
Summary
Store
reference-results-metadata.txtinside each archive's case directoryStop writing one shared metadata file per tutorial
Drop the SHA256 checksum from the metadata (it described the outer
.tar.gz, which no longer makes sense once the file lives inside that archive)Ignore the embedded metadata during field comparison
Rename the metadata file to
reference-results-metadata.txtso editors open it as plain textUpdate the system-test documentation
Closes #811
Test Plan
Packed sample archives locally and verified
reference-results-metadata.txtends up inside each case folder, with exports and iteration logs unchanged, and no shared metadata file written anymore. Also verified fieldcompare ignores the metadata file, and ran pre-commit on the changed files. Full regeneration of reference archives happens after merge, as discussed.